Add Integrity Checker for BookTitle #13854

TheYorouzoya · 2025-09-11T12:38:57Z

Closes #12271

This PR adds a new integrity checker for the booktitle field along with the associated cleanup actions.

I'll be doing the development in the following major phases (these will be refined further as I move along):

Gathering and defining a clear set of requirements
Drafting an implementation approach
Adding the core logic/classes required
Integrating the feature into the GUI
Iterate till feature meets expectations

Steps to test

Integrity Checker marks an improper Booktitle field

Check Integrity dialogue lists failing check for each embedded field individually

New booktitle cleanup checkbox and sub-panel added to Cleanup Entries Dialog Box

Clicking on the "Clean up 'booktitle'..." checkbox will enable the cleanup sub-panel, allowing the user to pick a cleanup action for each individual field found in booktitle

Post clean up, the fields are moved to their respective field editors

Mandatory checks

I own the copyright of the code submitted and I license it under the MIT license
I manually tested my changes in running JabRef (always required)
I added JUnit tests for changes (if applicable)
I added screenshots in the PR description (if change is visible to the user)
I described the change in CHANGELOG.md in a way that is understandable for the average user (if change is visible to the user)
[/] I checked the user documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request updating file(s) in https://github.com/JabRef/user-documentation/tree/main/en.

Part of JabRef#12271

TheYorouzoya · 2025-09-12T12:35:35Z

Requirements

1. Integrity Checker should flag a `booktitle` with year numbers, locations, and page numbers

The booktitle field should be marked similar to when an integrity check fails for other fields (for example, Author)

The field should also show up in the Check Integrity dialog box in Quality -> Check Integrity

2. Integrity Checker should allow the user to move nested fields to their appropriate place via a cleanup action

If a booktitle is found to contain years. locations, or page numbers, the user should be allowed to perform a cleanup action under Quality -> Clean Up Entries dialog box which would move them to their appropriate fields.

Since it is possible that the field we're trying to move our extracted data to is already populated, the user should be given a choice whether to perform the move for each piece of data found. This translates to adding three cleanup actions or options to the dialog box (one each for year, page number, and location).

Implementation Approach: Details, Edge Cases, and Questions

I'll take an example from the issue post and break it down as follows:

Input: European Conference on Circuit Theory and Design, {ECCTD} 2015, Trondheim, Norway, August 24-26, 2015
Year: 2015, 2015
Month: August
Page Numbers: 24-26
Locations: Trondheim, Norway

Year Numbers

Years can be of the form XXXX with four digits on their own. Since the booktitle field deals with scientific journals, books, and conferences, we can further refine this in the range 16XX to 20XX (the first ever publications are from the 1600s).

Q1. What about multiple years present in a title? Which one do we pick to transfer to the year field? My guess would be the latest one.

Months

Months are spelled out in text (like January, February, and so on). These can be easily picked up via a regex or a simple string comparison. Once found, they can be moved to the Month field under Optional fields.

Page Numbers

These will typically be of the form <number>-<number> (like 22-45) or <number>--<number>(22--45). There aren't any edge cases here to talk about other than whether we want to support more formats.

Note

I'm assuming that the 24-26 in the example is referring to a page range and not a date. If that is not the case, then I'd like some examples of page numbers in a booktitle for reference.

Locations

These can refer to the names of countries and/or cities present in the field. While a previous attempt doesn't provide much in terms of details, this comment on the original issue post does suggest an offline-friendly approach.

Since we'll encounter location names only in cases of conferences, we're only looking at cities big enough to host one of them. GeoNames provides multiple datasets with cutoffs based on population (>500, >1000, >5000, >15000). Just to be on the safe side, we can pick the >1000 population dataset and incorporate that into our search database.

The data has 162,090 cities in it along with a bunch of associated information. We'll strip away all the metadata and keep only the names. Since the data provides city names in UTF-8 as well as in ASCII as separate columns, we'll flatten it further down to one city per line and deduplicate the resulting dataset. Doing this brings us down to 1.9MB from our ~30MB starting point with the total number of cities going up to 173,371 (not that many cities have different UTF-8 and ASCII names). We can add the list of countries on top of this to get our final dataset.

Note

If we really want to be stingy about space, we can further compress this down using something like gzip and get down to around ~800KB as plaintext can be compressed quite well.

Loading this many entries into memory shouldn't be that big of burden either. A GPT-assisted rough calculation puts us at around 40MB of heap usage if we're using a HashSet<String> [Edit: I'm now reconsidering the HashSet and instead using a specialized Trie data structure to accommodate for punctuation and whitespace within location names].

Q2. Is this approach okay? If there are issues with the overhead, we can use a bloom filter, but that is a probabilistic data structure which can lead to some false positives.

Q3. Is there a specific field where these should be moved to as part of the cleanup action? I have noticed these fields: address, location, and venue.

TheYorouzoya · 2025-09-12T12:40:31Z

@koppor please check if the approach fits with the expectations of the feature, and help clarify those questions. I'll start laying out some of the core logic in the meantime.

TheYorouzoya · 2025-09-27T10:44:17Z

Since I have not received any feedback on the approach for the last two weeks, I'll be pushing an implementation as per the outlined approach in a couple of days.

- Add integrity checkers for each of the following fields: Year, Month, Page Range, and Location. - Update FieldCheckers to apply the checkers on the Booktitle, Journal, and Journaltitle fields. - Add location data (countries_cities1000.txt) containing country and city names. The data is sourced from - Add LocationDetector to load location data and allow location extraction from input. - Add BooktitleCleanupField enum to represent each checker field. - Add BooktitleCleanupAction enum to represent the following cleanup actions: remove only, replace, replace if empty, and skip. - Consolidate reused pre-compiled regex patterns into RegexPatterns. - Add BooktitleCleanupPanel and BooktitleCleanupPanelViewModel (inspired by FieldFormatterCleanups) to the GUI. - Add BooktitleCleanupPanel.fxml for the cleanup section's layout and integrate it into CleanupPresetPanel.fxml. - Update CleanupPreferences, CleanupPresetPanel, and CleanupWorker to include Booktitle cleanups. - Update module-info.java to export the BooktitleCleanupAction enum. - Add localization keys to JabRef_en.properties for all the new cleanup GUI text. - Add unit tests and relevant javadoc. Part of JabRef#12271

Part of JabRef#12271

jabgui/src/main/java/org/jabref/gui/commonfxcontrols/BooktitleCleanupPanel.java

jablib/src/main/java/org/jabref/logic/cleanup/BooktitleCleanups.java

jablib/src/test/java/org/jabref/logic/integrity/BooktitleLocationCheckerTest.java

jablib/src/test/java/org/jabref/logic/integrity/BooktitleMonthCheckerTest.java

TheYorouzoya · 2025-09-29T18:54:45Z

@koppor check if the feature implementation so far is up to expectations. Also, please help me figure out whether it requires adding a preference migration or updating preferences in the CliPreferences to have JabRef "remember" the previous choice for the cleanup panel.

jablib/src/test/java/org/jabref/logic/integrity/BooktitlePageRangeCheckerTest.java

jablib/src/test/java/org/jabref/logic/integrity/BooktitleYearCheckerTest.java

jablib/src/test/java/org/jabref/logic/util/LocationDetectorTest.java

Part of JabRef#12271

koppor · 2025-10-17T12:19:42Z

Also, please help me figure out whether it requires adding a preference migration or updating preferences in the CliPreferences to have JabRef "remember" the previous choice for the cleanup panel.

I assume the functionality is existing for the other settings. Thus, please also support this for the new checkbox. Thank you 😅.

TheYorouzoya · 2025-10-17T13:56:20Z

Also, please help me figure out whether it requires adding a preference migration or updating preferences in the CliPreferences to have JabRef "remember" the previous choice for the cleanup panel.

I assume the functionality is existing for the other settings. Thus, please also support this for the new checkbox. Thank you 😅.

My question wasn't a "whether or not" to add the functionality, it says "whether it requires A OR B" to be modified to have it enabled, i.e., which of the two options need to be updated to get it to work.

- Updated CleanupPresetPanel, BooktitleCleanupPanel, and BooktitleCleanupPanelViewModel bindings to have JabRef remember the previous cleanup action selection. - Updated CleanupAction to update the BooktitleCleanups from the current cleanup preset value. - Update BooktitleCleanups to return a map of the current cleanup actions. - Update CleanupPreferences to allow setting booktitleCleanups property directly. Part of JabRef#12271

Part of JabRef#12271

Siedlerchr · 2025-11-15T14:09:49Z

jablib/src/main/java/org/jabref/logic/cleanup/CleanupPreferences.java


    public CleanupPreferences(EnumSet<CleanupStep> activeJobs) {
-        this(activeJobs, new FieldFormatterCleanups(false, new ArrayList<>()));
+        this(activeJobs, new FieldFormatterCleanups(false, new ArrayList<>()), new BooktitleCleanups(false, LocationDetector.getInstance()));


This is the wrong approach, you have to use this CleanupStep and add it tot the list of Cleanup Steps
, see also CleanupWorker

Are you suggesting to remove the section which lets the user pick replacement options? That is, the second rectangle in this picture (same image as in the PR above)

If not, then how would you suggest managing the state for the currently selected options?

I remember that the CleanupStep enum is for simple cleanups that do not keep state. The booktitle cleanup can potentially replace the data in 4 other fields; which is why a move should be confirmed by the user beforehand. This requires keeping track of state across each of the 4 fields.

The only CleanupStep cleanup that has any state tied to the UI is this one:

This single extra checkbox state is represented as two entries within the enum:

public enum CleanupStep { ... RENAME_PDF, RENAME_PDF_ONLY_RELATIVE_PATHS, ...

And the CleanupWorker calls the constructor separately while adding the cleanup job:

return switch (action) { ... case RENAME_PDF -> new RenamePdfCleanup(false, () -> databaseContext, filePreferences); case RENAME_PDF_ONLY_RELATIVE_PATHS -> new RenamePdfCleanup(true, () -> databaseContext, filePreferences); ...

The same if-else chain exists on the GUI side in the CleanupPresetPanel:

if (cleanUpRenamePDF.isSelected()) { if (cleanUpRenamePDFonlyRelativePaths.isSelected()) { activeJobs.add(CleanupPreferences.CleanupStep.RENAME_PDF_ONLY_RELATIVE_PATHS); } else { activeJobs.add(CleanupPreferences.CleanupStep.RENAME_PDF); } }

Adding the booktitle cleanup as a CleanupStep enum entry would require chaining similar if-else statements, but even longer (not to mention adding half a dozen entries to the enum). This is why I chose to model the cleanup after the FieldFormatterCleanup, which is also a modular cleanup that tracks state.

Clarify if this is what you want and I'll go ahead and implement it that way.

I didn't see that you added a user choice in the UI... Not sure where the motivation comes from and how the user should decide.

Compare to the cleanup of the notes-to-doi field - it just leaves the note as is if there is already a doi. However, if the DOI is equal to the one in the notes field, the doi is removed from the notes field.

The booktitle cleanup can to the same for each extracted field.

More sophisticated it would be to show a merge dialog with the two options and then the user can decide. In the current implementation, the user should know which field is correct, which they cannot see just looking at the dialog. -- In other words: We need to think from the user side, not from the algorithmitc side.

See org.jabref.logic.cleanup.DoiCleanup for the code.

I would hope your cleanup is doing simlar things. I think, it should just be a CleanupStep without more user interaction.

I didn't see that you added a user choice in the UI...

The user decides, for each field:

Remove Only: Clean up the booktitle field without moving the extracted value.

Replace: Move the extracted value to its target field and overwrite any existing value.

Move If Empty: Move the extracted value only if the target field is empty.

Skip Cleanup: Skip the cleanup altogether; in case a field value (like month or location) is actually part of the booktitle.

Hopefully this clarifies how the user choices are implemented.

Not sure where the motivation comes from

The motivation comes from the original issue (screenshot attached). The idea was to give users control in cases where the extraction may or may not be desired.

Compare to the cleanup... The booktitle cleanup can to the same for each extracted field.

This is handled through option 3 above (“Move If Empty”). That effectively mirrors how the general cleanup behaves.

More sophisticated it would be to show a merge dialog...

This sounds like a good enhancement. We can also add an option to apply a default cleanup selection automatically in case a user doesn't want to go through each entry manually (similar to how a copy-paste dialog box works on windows).

I think, it should just be a CleanupStep without more user interaction.

Just to confirm: do you mean defaulting to option 3 (“Move If Empty”), removing the radio-button UI entirely, and moving the logic into the CleanupStep enum? I just want to make sure I’m interpreting your suggestion correctly before proceeding.

Just to confirm: do you mean defaulting to option 3 (“Move If Empty”), removing the radio-button UI entirely, and moving the logic into the CleanupStep enum? I just want to make sure I’m interpreting your suggestion correctly before proceeding.

Yes.

Longer answer:

Hopefully this clarifies how the user choices are implemented.

How should the user know what to choose!

Is there a general answer or is it a case-by-case basis?

How to distinguish the cases?

Can there be more automation?

The logic in the note-cleanup shows a fully-automated approach - and the user doe snot need to decide anything.

When it comes to the mentioned "confirmation", its time for the merge dialog

Siedlerchr · 2025-11-15T14:10:07Z

jablib/src/main/java/org/jabref/logic/cleanup/CleanupWorker.java

            jobs.addAll(preset.getFieldFormatterCleanups().getConfiguredActions());
        }

+        if (preset.getBooktitleCleanups().isEnabled()) {


not needed if you put intot he cleanup steps

Siedlerchr · 2025-11-15T14:17:35Z

jablib/src/main/java/org/jabref/logic/util/LocationDetector.java

+    private int maxLocationLength = 0;
+    private int minLocationLength = Integer.MAX_VALUE;
+
+    private LocationDetector() throws JabRefException {


This whole class should be loading the data from an mv file like for example we do it with nournal abbrvatiosn or other data

I'll look into it.

koppor · 2025-11-18T19:17:25Z

Also, please help me figure out whether it requires adding a preference migration or updating preferences in the CliPreferences to have JabRef "remember" the previous choice for the cleanup panel.

I assume the functionality is existing for the other settings. Thus, please also support this for the new checkbox. Thank you 😅.

My question wasn't a "whether or not" to add the functionality, it says "whether it requires A OR B" to be modified to have it enabled, i.e., which of the two options need to be updated to get it to work.

I think, its both. B to have it working, A to update existing preferences to have it enabled for all users.

koppor · 2025-11-18T19:22:13Z

Since I have not received any feedback on the approach for the last two weeks,

Sorry for this. We do this in our freetime and have not enough freetime to cover all contributions. We try to invest more - currently by skipping sports and reducing our sleep. Since a day has only 24 hours, also there are limits.

TheYorouzoya · 2025-11-20T14:18:41Z

Sorry for this. We do this in our freetime and have not enough freetime to cover all contributions. We try to invest more - currently by skipping sports and reducing our sleep. Since a day has only 24 hours, also there are limits.

Thanks for the update. I completely understand the time constraints. Open source is voluntary and I appreciate the work.

Just to clarify the context of my earlier ping: I wasn’t asking for a full code review (there was no code in the PR at that time). I was hoping to confirm the implementation approach before investing more time in writing code. Because this issue has had discussions spread across several years, it helps to re-establish common ground regarding the expectations and the architecture up front.

As for code, the last message I sent was on September 30th, 50 days ago, to ask for help when I was stuck. That has now been answered.

koppor · 2025-11-20T15:53:36Z

As for code, the last message I sent was on September 30th, 50 days ago, to ask for help when I was stuck. That has now been answered.

Sure. We also might have missed some messages in the dev chat. It is very OK to ask after a few weeks. It is also very OK to look into other PRs and try to give feedback to other contributors. Some need help with their submodules; this is what most contributors should be able to support. By that, our community grows, contributors get more feedback and it also decreases the delay of answers at the own project, because the whole community spends more time on the project.

koppor · 2025-11-20T15:58:28Z

@TheYorouzoya Please not that we merged a PR started September, 11 also touching the GUI (#13852). Maybe, you first remove the radio buttons and then try to merge latest main to resolve the merge conflicts. -- You could also split-up the PR into two: One for the integrity check and one for the GUI. Some of us have success with GitButler, but this tool also needs some time to get used to it. Its benefit its: one sees the changes of all PRs combined - and can assign local changes to PRs. -- You can also "just" finish the PR,

The license of jablib/src/main/resources/util/countries_cities1000.txt needs to be clear. It would be OK if it had a different license; however it should be permissive and not copyleft. MIT would be best.

Since we also asked for an MVStore for that, maybe, a separate PR should start with that... Because this is deep in our build pipeline and you need to get used to JBang etc.

koppor · 2025-11-20T16:57:33Z

What I meant: move if empty, if equal: remove, else noop

Update Changelog

2528b8c

Part of JabRef#12271

TheYorouzoya and others added 3 commits September 29, 2025 22:25

Merge branch 'main' into booktitle-integrity-checker

4332291

Fix Whitespace Formatting Issues Across Files

8f0a25f

Part of JabRef#12271